Neural Information Processing Systems http://nips.cc/

Recently, strong results have been demonstrated by Deep Recurrent Neural Networks on natural language transduction problems. In this paper we explore the representational power of these models using synthetic grammars designed to exhibit phenomena similar to those found in real transduction problems such as machine translation. These experiments lead us to propose new memory-based recurrent networks that implement continuously differentiable analogues of traditional data structures such as Stacks, Queues, and DeQues. We show that these architectures exhibit superior generalisation performance to Deep RNNs and are often able to learn the underlying generating algorithms in our transduction experiments.

Memory has a pivotal role in human cognition and many different types are well known and intensively studied[1]. In neural networks and signal processing the use of memory is concentrated in preserving in some form (by storing past samples or using a state model) the information from the past. A system is said to include memory if the system's output is a function of the current and past samples. Feedforward neural networks are memoryless, but the time delay neural network [2], the gamma neural model [3] and recurrent neural networks are memory networks. An important theoretical result showed that these networks are universal in the space of myopic functions [4]. A methodology to quantify linear memories was presented in [3], which proposed an analytic expression for the compromise between memory depth (how much the past is remembered) and memory resolution (how specifically the system remembers a past event). A similar compromise exists for nonlinear dynamic memories (i.e. using nonlinear state variables to represent the past), but is depends on the type of nonlinearity and there is no known close form solution. It is fair to say that currently the most utilized neural memory is the recurrent neural networks (RNN) for sequence learning. Compared to the time delay neural network, RNN keeps a processed version of the past signal in its state.

For more on derivatives and differentiability, see the rest of that tutorial.) Why do we care that the stack (as a function) is differentiable? Well, we used the "derivative" of the function to move the error around (more specifically... to backpropagate). For more on this, please see the Tutorial I Wrote on Basic Neural Networks, Gradient Descent, and Recurrent Neural Networks. I particularly recommend the last one because it demontrates backpropgating through somewhat more arbitrary vector operations... kindof like what we're going to do here.

For more on derivatives and differentiability, see the rest of that tutorial.) Why do we care that the stack (as a function) is differentiable? Well, we used the "derivative" of the function to move the error around (more specifically... to backpropagate). For more on this, please see the Tutorial I Wrote on Basic Neural Networks, Gradient Descent, and Recurrent Neural Networks. I particularly recommend the last one because it demontrates backpropgating through somewhat more arbitrary vector operations... kindof like what we're going to do here.

Recently, strong results have been demonstrated by Deep Recurrent Neural Networks on natural language transduction problems. In this paper we explore the representational power of these models using synthetic grammars designed to exhibit phenomena similar to those found in real transduction problems such as machine translation. These experiments lead us to propose new memory-based recurrent networks that implement continuously differentiable analogues of traditional data structures such as Stacks, Queues, and DeQues. We show that these architectures exhibit superior generalisation performance to Deep RNNs and are often able to learn the underlying generating algorithms in our transduction experiments.